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Abstract 

Real-time tracking of multiple targets or geometrical 
features of an object using a variable resolution laser 
scanner is presented. The scanner is characterized by 
its robustness to ambiant illumination, even the sun 
shining directly into the sensor, and its tracking 
resolution. The sensor to extract registered range and 
intensity information for each scanned point on the 
object at a rate of 18 kHz uses two high-speed 
galvanometers and a collimated laser beam. Three- 
dimensional real-time tracking using Lissajous 
patterns proves to be very attractive for space 
applications. Integration with the existing 
photogrammetry-based Advanced Space Vision 
System (ASVS) is discussed. 


Introduction 

It is now well accepted that vision will play a major 
role in both supervised and unsupervised operations 
for the automation of several space-related activities. 
Specifications state that the Artificial Vision Unit will 
support rendez-vous and proximity operations 
including payload tracking, capture, and berthing in 
both teleoperation and adaptive control modes. 12 It is 
also recognized that vision will play a major role 
during the assembly and maintenance operations of 
the space station. 

The current Artificial Vision Function (AVF) is based 
on the Space Vision System (SVS) that was 


demonstrated on CANEX-2. The SVS analyzes video 
signals from the closed circuit television system of 
the space shuttle and provides real-time position and 
orientation information about an object with a 
cooperative target array. 2. The baseline requirements 
for the Artificial Vision Function are as follows: 

- To identify a suitably illuminated object (target) 
from its video image. 

- To estimate the position, attitude, translation, and 
rotational rate of the object. 

- To provide appropriate camera control to track 
the object. 

- To be able to track objects before capture by 
manipulators and berth objects/payloads handled 
by manipulators. 

To achieve robust adaptive control, supervision, and 
inspection, the vision system must be 100% 
operational throughout the changing illumination 
conditions in orbit. Unfortunately, the quality of the 
images produced by standard video-camera-based 
systems is adversely affected by the presence of the 
sun or any other strong source of light. Poor contrast 
between features on the object and background, and 
saturation of the photo-detector array often make it 
difficult to analyze the video images. Video camera 
reliability during normal operation is questionable and 
can compromise the success of the operation or at 
least seriously limit its practical use. 

Although camera-based systems are very attractive 
because of their ease of use and simplicity of 
integration with existing equipment, it is highly 
desirable to offer a complementary vision system that 
will not be restricted by operational conditions such 
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as sun interference. The proposed laser scanner 
approach offers the advantage of being almost 100% 
operational throughout the changing illumination 
conditions in orbit. The technique, developed at NRC 
in collaboration with the CSA, is designed to be 
insensitive to background illumination such as the 
earth albedo and the sun radiation and most of its 
reflections. 3,4 Immunity to background interference is 
the result of the very narrow band of frequencies 
emitted by the laser light that can be tuned by optical 
filters, by special optical design to extract range 
information, and by proprietary real-time signal 
processing techniques used to distinguish between the 
laser beam and any remaining sources of interference 
(e.g., specular reflections). 

The Laser Scanner 

Figure 1 shows the geometry of this prototype based 
on the auto-synchronized time-flying single-point 
technique. 5 The system is comprised of two 
orthogonally mounted scanning mirrors: the X-axis 
scanning mirror, used for range triangulation, and the 
Y-axis mirror. The two mirrors are driven by accurate 
galvanometers. The light beam generated by the laser 
is deflected by a mirror and scanned on the object. A 
camera, consisting of a lens and a position-sensitive 
photodetector, measures the location of the image 



Figure 1: Schematic representation of the dual- 
axis auto-synchronized range scanner (reprinted 
from Reference 7). 


illuminated point on the object. By simple 
trigonometry, the three-dimensional coordinates of 
that point are calculated. 6,7 

Some of the advantages of the auto-synchronized 
technique are a large field of view, high accuracy, 
and immunity to ambiant illumination. 8 A 
conventional camera must continuously monitor the 
whole field of view, making it very susceptible to sun 
interference. The laser scanner technique has a small 
instantaneous field of view limiting the undesired 
interference. Such a system is optically light efficient 
because the received laser power is focused onto a 
small spot thereby increasing the signal-to-noise ratio 
of the returned signal. Automatic control of the laser 
power source further extends the dynamic range of 
the scanner. 

Figure 2 illustrates the equivalent optical geometry. 
The basic concept is that the projection of the light 
spot is synchronized with its detection. The 
instantaneous field of view of the position sensor 
follows the spot as it scans the scene. An external 
optical perturbation can interfere with the detection 
only when it intercepts the instantaneous field of 
view of the scanner. At this level, electronic signal 
processing is used to filter these false readings to 
obtain the correct 3-D measurement. The total field of 
view of the laser camera is related only to the 
scanning angles of the galvanometers and mirrors as 
opposed to a conventional camera where field of view 
and image resolution are intimately linked, the larger 
field of view the smaller pixel resolution achievable. 



Figure 2: Basic principle of synchronized 
scanner, second axis not shown (reprinted from 
Reference 8 ). 
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Although laser scanners are usually slower than their 
TV camera counterparts when used in a conventional 
imaging or raster scan mode of operation, the same 
laser scanner can obtain refresh rates of more than 
130 Hz with a pointing resolution of 15000 elements 
x 1 5000 elements in the tracking mode (single target). 
Consequently, the resolution and speed of the laser 
scanner exceed conventional video cameras. The high 
pointing accuracy of a laser beam and the large depth 
of field of the scanner enable the acquisition of large 
3-D images of 4000 pixels x 4000 pixels or more. To 
achieve similar resolution, a video camera would 
need a high-quality optical lens and specially 
designed high-resolution CCD cameras. 

A custom-designed galvanometer controller board 
gives access to any pixel in the field of view of the 
range sensor and provides the random access tracking 
capability. 3,4 The controller permits interrogation of 
any point in the scene without having to sample the 
entire field of view of the scanner. This Region Of 
Interest (ROI) sampling technique offers great 
potential for rapid and efficient acquisition of dense 
and accurate 3-D images over a large volume of 
measurement. 

Figure 3 is a photograph of the laboratory prototype 
of the auto-synchronized laser scanner developed for 



Figure 3: Photograph of the laser scanner 
prototype. 


this application. Tables I and II and Figures 4 and 5 
summarize the characteristics of this laser scanner 
prototype in the triangulation mode of operation. 



Figure 4: Range resolution (single point). 



Figure 5: Angular pointing resolution (tracking 
mode using Lissajous figures). 


Real-time Tracking 


Real-time tracking of targets or geometrical features 
on an object is implemented using Lissajous figures, 
to obtain good scanning speed and accuracy 3 A 
Lissajous figure is mathematically defined by 

0 ( t) ^cos (/no) t+ 6 0 ) + 0 O (1) 

<t> ( t) =A^cos (nco t+ + <fr 0 (2) 

where A Q and are the amplitudes of the Lissajous 
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Table I: Physical characteristics. 


Dimensions 

10.5 in X 6.25 in x 4 in 

Data Type 

3-D and Intensity 

Field of view 

30° x 30° 

Working range 

0.6 ft to TBD (>150 ft) 

Range accuracy 

! See Figure 4 

Acquisition Speed 

1 18 kHz 

Scanning System 

j Galvanometers 

Resolution X-Y 

| 1 / 15000 


pattern, (0 the refresh rate of the scan, 8 0 and 8* the 
relative phases of the sinewaves, t the sampling time, 
and 0 O and <|> 0 the position of the center Lissajous 
pattern. The pattern shape is defined using parameters 
m:n. For example the pattern illustrated in Figures 6 
and 7 is a 3:2 Lissajous figure. 


Lissajous 

Scan 



Tracking 

Error 


Center 
of target 


Figure 6: Tracking of a circular target using a 3:2 
Lissajous pattern. 


Lissajous patterns are used to scan objects at refresh 
rates exceeding the frequency response of the 
mechanical deflection system. The scanning device is 
not required to stop the acquisition after each trace 
line as opposed to the raster-type scan illustrated in 
Figure 8. Instead, the full pattern is used to acquire 
both range and intensity information. Furthermore, 
the natural inertia of the galvanometer-mirror 
structure smooths the scanning pattern and hence 
increases the pointing accuracy of the tracking 



Figure 7: Real-time tracking of geometrical targets 
(e.g., corner or targets) using Lissajous patterns 
and the laser range sensor. 



Figure 8: Conventional raster-type acquisition of 
the 3-D shape of the object. 

system. As well, Lissajous pattern signals are 
optimally filtered using the Fourier transform (or 
notch filter), thus reducing electrical noise, 
quantization effects, and distortions. 9 

Tracking is implemented using the 3-D range 
information on the Lissajous pattern, the returned 
intensity signal from the object, or a combination of 
both. For example, the location and orientation of the 
geometrical target is obtained with either the 3-D or 
intensity edges of the object or the intersection of the 
measured surfaces on an object. 
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Table II: Performances characteristics of the laser scanner. 


Laser Scanner 


Video Cameras 


General 

- Excellent immunity to ambiant illumination 

- High-resolution images and large depth of view 

- 3-D and registered intensity images 

- New technology not as widely used 

Optics 

- Optically efficient, excellent signal-to-noise ratio 
because all the laser energy is concentrated in a 
single point 

- Simple on-axis optics 

- Large depth of view (max/min range) and small 
instantaneous field of view 

- Optical magnification produces good range 
resolution and large images of the sun to simplify 
the signal processing and reduce saturation 

- Laser light easily filtered using narrow-bandwith 
optical interference filters 

- Eye safety requirements for terrestrial applications 
(eye-safe if laser at 1.5 jim is used) 

Scanning devices (galvanometers) 

- Large scanning angles, high pointing resolution 

- Randomly addressable, almost any scanning 
patterns can be programmed 

- Mechanically moving devices (relatively slow) 

- Temperature sensitive* 

Electronics 

- Simplify the processing of the CCD image 

- Further improve the immunity to ambiant 
illumination 

- Modular architecture easily upgradable 

- Random control and real-time implementation 
requires good knowledge of the dynamics of the 
sensor and the application** 


- Highly susceptible to ambiant illumination 

- Low hardware cost 

- Use on-board closed-circuit video cameras system 


- Simple camera configuration 

- Work typically under ambiant illumination or 
projectors 

- Adversely affected by ambiant illumination and sun 
interference 

- Large instantaneous field of view makes the system 
susceptible to saturation 

- No optical filtering possible because of the use of 
ambiant illumination (if laser projectors are used 
large-bandwith optical filters are required) 

- Compromise between optical magnification and 
field of view (focal length) and sensitivity and 
depth of view (f-number) 


- No moving parts except if rotation stage is used for 
camera pointing 

- Mechanically rugged systems 

- Temperature sensitivity is reduced by years of 
design and mature technology 


- 2-D image processing required at video rate 

- Special technique still to be developed to reduce 
the effect of changing illumination on the targets 

- Modular low-cost electronics 

- All solid-state 


* Temperature compensation is implemented using synchronization photo-detectors. K 

** This is valid for any accurate measurement system. 
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Figure 6 illustrates the tracking principle using a 
Lissajous figure. The target can be a cylinder, a 
sphere, or simply a colored target . For long-range 
measurement a retroreflective target (corner cube or 
3M retroreflective tape) is used. Assuming that /(/) 
and Z(t) are, respectively, the intensity and range 
measurements for each point of the Lissajous pattern 
defined by equations 1 and 2, the tracking error on 
the target is computed from the centroid 
measurements weighted by the intensity of the 
returned laser signal: 

i v-i (t)= Ht)-i Z9f 

if (I(t)>I Z9f ) and (3) 
(Ar min < (r(t) -r 0 ) <Az aax ) 

= 0 otherwise 


$= H diz *A$+H pz9d +9 pz9d 

+H 9dj *A9 adj ( 8 ) 

5= [r,0,4>] 


a„+a 

l-piZ^-PjZ ' 2 




1 

l-(l-Y) z -1 


( 10 ) 


H *V {Z) = < C<1 (11 > 


A Io 


(4) 


A8-£ e(t) i^ (t) - 0 O 

E J v.i < t) 


(5) 


A ,. £ii£>W£L - * 


(6) 


If a target is lost then 

A r= A0 = A<J> =0 

if (E J v.i(t)=o) 


(7) 


0 O , ^ 0 ’ r o the position of the scan and consequently 
the expected position of the target. The intensity 
threshold / rcf and range validation window r min and 
r max are specific for a target type. Any point outside 
the range/intensity window is considered not valid 
and disregarded. Range is extremely useful in 
equation 3 to make the technique very robust to 
external perturbations (e.g., r=°o for the sun) 


z‘ ] is the z-transform delay of one sample. // dr is the 
direct feedback loop internal to the laser scanner, 
H d is the predicted or expected position of the 
target, and is an external user control 

adjustement. // dr implements a classical deadbeat 
controller reducing the tracking error to zero such that 
the Lissajous pattern is always centered on the target. 
The predicted or expected position is used to 
give an estimate of the position of the target. This 
estimate is obtained either from the known or 
calculated position of the object or from the ASVS- 
laser scanner interface. allows an immediate fine 
adjustment of the target position. This is especially 
useful in the search mode when the target is not 
found (equation 7) and the predicted position is not 
accurate enough (e.g., during initial search) to lock 
the laser scanner pattern on the target. External user 
control is used to move the pattern over the target. 
Equation 11, a lossy integrator, gradually removes 
this external correction and is replaced by the 
correction H$ r 

The laser scanner automatically adjusts the size of the 
scan according to the distance of the target from the 
camera: 





( 12 ) 


Tracking is implemented in the spherical coordinate 
system (r,0,<|)) of the laser scanner using the digital 
feedback structure defined, using the z-transform by: 


For each target, a set of parameters A I K{ , Ar min ,Ar majt 
is defined. Optimum coverage of the surface of the 
target and pointing resolution are obtained because 
the Lissajous pattern is always centered and scaled on 
the target independently of the distance of the target. 
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Multiple Tracking of Targets, Photo gram me try, 
and the Laser Scanner 

The photogrammetry mode of operation of the ASVS 
to be used on board the space shuttle 1 and the laser 
scanner are compatible and easily interfaced; the 
deflection angles of the galvanometers directly 
provide the angular separation between the targets. 
Then, with photogrammetry techniques, the object 
position X-Y-Z and attitude parameters yaw-pitch- 
rotation are computed by minimizing the quadratic 
error between the expected position of the targets 
computed from the model of the object and the 
measured positions provided by the laser scanner. 2 

Real-time and physical constraints caused by 
switching quickly between multiple targets and 
limited by the inertia of moving mechanical devices 
must be considered, based on the application 
requirements. A careful investigation of the complete 
mechanical, optical, and electrical characteristics of 
the hardware and software is needed to achieve 
optimum tracking performance. 

The laser scanner is programmed to sequentially scan 
different sections or targets on one or multiple 
objects, as illustrated in Figures 7 and 9. A list of all 
possible scanning strategies used for a given task are 
pre-defined. Each strategy contains a list of the 
visible targets to scan and is optimized for a given 
application. The laser scanner uses this list to 
sequentially scan the targets on the different objects. 

A scenario under study for the assembly of the space 
station is the berthing of the LAB/A Module on the 
APN (Aft Port Node) CBM interface in the Space 
Station using the Remote Manipulator System (RMS) 
as graphically illustrated in Figure 9. The vision 
system task is to provide the position and orientation 
of the LAB/A module relative to the APN and ITA 
(Integrated Trust Assembly, not shown) modules as 
well as visual guidance during berthing operations. 

The visible and nonvisible targets on the two main 
objects are: 

Targets Module 

T, to T x LAB/A 

T 9 to T 16 APN (Aft Port Node) 

Only the visible targets T, to T 4 and T 9 to T, 2 are 
shown in Figure 9. 



Figure 9: Example of tracking for the assembly of 
the space station. 

The following strategies are defined for this visible 
section of the objects: 

Strategy = Scanning Target List 

S 0 = T|,T 2 ,T 3 ,T 4 

s, = T 9 ,T| 0 ,T„,T I2 

s 2 = So, T 9 , So, T,o, So, T n , S 0 , T 12 

It is important to acquire fast moving targets as often 

as possible because of the sequential nature of the 
scanning method. The basic scanning strategies S 0 and 
S, scan oniy the targets on their respective objects. 
They are used to initialy locate a relatively fast 
moving object. When the object is locked by the 
scanner, the scanning strategy is switched to S 2 to 
measure the relative position of the LAB/A module 
with respect to the whole structure. In this example 
the APN module is considered to be almost stationary 
and therefore S 2 instructs the laser scanner to scan the 
moving LAB/A module more often than the other 
objects. All the targets from the LAB/A module are 
scanned (S 0 ) followed by one target from the APN 
module. Although only three strategies are shown, 
any number can be defined, depending on the 
application. 

The approach used for object tracking allows the 
operator to fully define the tasks required by the 
tracking system. Instead of having a single "target or 
object" detection algorithm controlling the scanner, 
the user can dynamically tune or replace any element 
for a given application. New modules are easily built 
to improve system performances. 
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The tracking refresh rate for the LAB/A module is 
currently limited to 15 Hz for this experiment. 
Geometrical detection and tracking are 
computationaly expensive and need more processing 
power to achieve faster tracking speed than the 
current single 68020 VME processor can provide. 
Integration of parallel processing using multiple 
TMS320C40 DSP is planned. Furthermore, several 
key elements of this system, including geometrical 
target localization algorithms, absolute real-time 
calibration (vs. relative calibration), and optimized 
corrector-predictor tracking techniques, are only at 
their preliminary stage of study. As the system 
improves in performance, further modifications can 
be introduced without redesigning the whole software 
or hardware environment. 


Laser Scanner and Three-dimensional Imageiy 

A more conventional imagery method can also be 
used for tracking, at lower speed than the Lissajous 
technique. Three-dimensional imagery using a laser 
scanner has the major advantage of creating high 
resolution images of the object under inspection. 
Images of more than 4000 points x 4000 points are 
easily acquired Figure 8 illustrates the raster-type 
acquisition mode, similar to conventional video 
cameras except that 3-D range information and 
intensity, in perfect registration, are obtained for all 
the points scanned in the image. Resolution is also 
higher than with conventional CCD cameras, as 
discussed previously. It is usually assumed that an 
object is relatively stationary with respect to the 
scanner. 

Figure 10 shows a 3-D raster image of a copy of the 
CTA testing module used during testing of the SVS 
system on board Space Shuttle Flight STS-52. It is 
used here for evaluation of geometrical tracking 
algorithms. Other scanning methods have also been 
proposed based on the random access laser scanner 
discussed here.' 0 


Conclusion 

Real-time tracking of targets on an object and 
acquisition of high-density three-dimensional images 
have been demonstrated using a random access 3-D 
laser scanner. The prototype has the potential of 
automatically tracking, in three dimensions, either 
geometrical features of the object itself or targets 



Figure 10: Raster type image of a model of the 
CTA used during space shuttle flight STS-52: 

(top) intensity image, (middle) range image 
encoded in grey levels , (bottom) pseudo shaded 
(derivative) of the range image (texture). 

attached to it. Excellent immunity to ambient 
illumination and sun interference is obtained. 

Real-time tracking of targets on an object is realized 
using Lissajous-type scanning patterns or raster type 
images, depending on the speed of the moving obje 
ct. The Lissajous scanning figures give high pointing 
resolution, excellent stability, and good object 
position refresh rate. 




Tracking error feedback is simultaneously obtained 
from three different sources: (1) direct tracking error 
based on each individual targets, (2) global predicted 
target position based on the calculated estimate of the 
object position, and (3) external feedback control 
from the human operator. Tracking of multiple targets 
using Lissajous patterns is based on user defined 
scanning strategies pre-programmed in the laser 
scanner according to the application requirements. 

Angular tracking resolution of 20 arcsec RMS at 150 
ft, for afield of view of 30° was measured (1/15000). 
A more complete study of the calibration parameters 
and of temperature variations on the laser scanner is 
still required to fully characterize the exact 
performance of the sensor. To obtain an absolute 
accuracy measure for the full working volume (160 
00 yd^), the difficult problem of obtaining a reference 
system more accurate than the laser scanner must be 
solved. Only resolution is quoted for the moment. 
Full calibration of the laser scanner/ASVS system 
combination is currently in progress. 
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